{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hello, World!\n",
    "\n",
    "This is our Hello, World! example based on building an error log analyzer using Fenic's semantic extraction capabilities to parse and analyze application errors without regex patterns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial Setup\n",
    "\n",
    "First we take care of our imports and configure our Session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fenic as fc\n",
    "\n",
    "# Configure session with semantic capabilities\n",
    "config = fc.SessionConfig(\n",
    "    app_name=\"hello_debug\",\n",
    "        semantic=fc.SemanticConfig(\n",
    "            language_models={\n",
    "                \"mini\" : fc.OpenAIModelConfig(\n",
    "                    model_name=\"gpt-4o-mini\",  # Fast and effective for log analysis\n",
    "                    rpm=500,\n",
    "                    tpm=200_000\n",
    "                )\n",
    "            }\n",
    "        )\n",
    "    )\n",
    "\n",
    "# Create session\n",
    "session = fc.Session.get_or_create(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the Data\n",
    "\n",
    "This dataset contains a collection of synthetic but realistic error logs from various microservices in a modern application stack. \n",
    "\n",
    "Each entry includes a timestamp, the service name, and a detailed error log message. \n",
    "\n",
    "The logs cover a range of issues such as exceptions, connection failures, timeouts, cache misses, slow queries, and more. \n",
    "\n",
    "This dataset is ideal for demonstrating error analysis, root cause extraction, and automated debugging workflows using semantic data tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " # 2. Create sample error logs - the kind developers see every day\n",
    "error_logs = [\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:23:45\",\n",
    "            \"service\": \"api-gateway\",\n",
    "            \"error_log\": \"\"\"\n",
    "ERROR: NullPointerException in UserService.getProfile()\n",
    "    at com.app.UserService.getProfile(UserService.java:45)\n",
    "    at com.app.ApiController.handleRequest(ApiController.java:123)\n",
    "    at java.base/java.lang.Thread.run(Thread.java:834)\n",
    "\n",
    "User ID: 12345 was not found in cache, attempted DB lookup returned null\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:24:12\",\n",
    "            \"service\": \"auth-service\",\n",
    "            \"error_log\": \"\"\"\n",
    "node:internal/process/promises:288\n",
    "            triggerUncaughtException(err, true /* fromPromise */);\n",
    "            ^\n",
    "\n",
    "Error: connect ECONNREFUSED 127.0.0.1:6379\n",
    "    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)\n",
    "    at Protocol._enqueue (/app/node_modules/redis/lib/redis.js:458:48)\n",
    "    at Protocol._write (/app/node_modules/redis/lib/redis.js:326:10)\n",
    "\n",
    "Redis connection failed during session validation\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:25:33\",\n",
    "            \"service\": \"payment-processor\",\n",
    "            \"error_log\": \"\"\"\n",
    "Traceback (most recent call last):\n",
    "  File \"/app/payment/processor.py\", line 89, in process_payment\n",
    "    response = stripe.Charge.create(\n",
    "  File \"/usr/local/lib/python3.9/site-packages/stripe/api_resources/charge.py\", line 45, in create\n",
    "    return cls._static_request(\"post\", cls.class_url(), params=params)\n",
    "  File \"/usr/local/lib/python3.9/site-packages/stripe/api_requestor.py\", line 234, in request\n",
    "    raise error.APIConnectionError(msg)\n",
    "stripe.error.APIConnectionError: Connection error: timeout after 30s\n",
    "\n",
    "Payment processing failed for order_id: ORD-789456\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:26:01\",\n",
    "            \"service\": \"data-pipeline\",\n",
    "            \"error_log\": \"\"\"\n",
    "django.db.utils.OperationalError: could not connect to server: Connection refused\n",
    "    Is the server running on host \"db.prod.internal\" (10.0.1.50) and accepting\n",
    "    TCP/IP connections on port 5432?\n",
    "\n",
    "FATAL: Batch job 'daily_analytics' failed after 3 retries\n",
    "Table 'user_metrics' has 2.3M pending records\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:27:15\",\n",
    "            \"service\": \"frontend\",\n",
    "            \"error_log\": \"\"\"\n",
    "TypeError: Cannot read property 'map' of undefined\n",
    "    at ProfileList (ProfileList.jsx:34:19)\n",
    "    at renderWithHooks (react-dom.development.js:14985:18)\n",
    "    at updateFunctionComponent (react-dom.development.js:17356:20)\n",
    "\n",
    "API response was: {\"error\": \"rate_limit_exceeded\", \"retry_after\": 60}\n",
    "Component tried to render before data loaded\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:28:03\",\n",
    "            \"service\": \"api-gateway\",\n",
    "            \"error_log\": \"\"\"\n",
    "WARN: Slow query detected in UserService.searchUsers()\n",
    "Query took 2.3 seconds to complete\n",
    "SELECT * FROM users WHERE name LIKE '%john%' ORDER BY created_at DESC\n",
    "Consider adding an index on the name column for better performance\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:28:45\",\n",
    "            \"service\": \"cache-service\",\n",
    "            \"error_log\": \"\"\"\n",
    "INFO: Cache miss for key 'user_preferences_12345'\n",
    "Fetching data from primary database\n",
    "Cache hit ratio: 87% (normal range: 85-95%)\n",
    "No action required\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:29:12\",\n",
    "            \"service\": \"notification-service\",\n",
    "            \"error_log\": \"\"\"\n",
    "WARN: Email delivery delayed for notification_id: notify_789\n",
    "SMTP server response: 450 Requested mail action not taken: mailbox unavailable\n",
    "Will retry in 5 minutes (attempt 2/3)\n",
    "            \"\"\"\n",
    "        },\n",
    "        {\n",
    "            \"timestamp\": \"2024-01-20 14:29:33\",\n",
    "            \"service\": \"analytics\",\n",
    "            \"error_log\": \"\"\"\n",
    "DEBUG: Processing batch of 1,250 events\n",
    "Memory usage: 45MB (limit: 512MB)\n",
    "Processing time: 1.2s\n",
    "All events processed successfully\n",
    "            \"\"\"\n",
    "        }\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Data\n",
    "\n",
    "Next we turn the data into a Fenic dataframe so we can start working with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create DataFrame from the error logs\n",
    "df = session.create_dataframe(error_logs)\n",
    "\n",
    "print(\"Hello World! Error Log Analyzer\")\n",
    "print(\"=\" * 70)\n",
    "print(f\"Found {df.count()} log entries to analyze\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define extraction schemas\n",
    "\n",
    "In this step we are keeping track of the schemas we want to use for extracting information from the data we have. Think of this as the equivalent of \"tool calling\" for LLMs but for data.\n",
    "\n",
    "Practically, Fenic ensures that whatever data the LLM will extract will be in the schema you provide. Also, the 'name' and 'description' of the fields, are used to provide context for the LLM on what to extract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create schema for extracting error analysis information\n",
    "error_analysis_schema = fc.ExtractSchema([\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"root_cause\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"The root cause of this error\"\n",
    "    ),\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"fix_recommendation\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"How to fix this error\"\n",
    "    )\n",
    "])\n",
    "\n",
    "# Create schema for extracting error type and component\n",
    "error_pattern_schema = fc.ExtractSchema([\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"error_type\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"Type of error (e.g., NullPointer, Timeout, ConnectionRefused)\"\n",
    "    ),\n",
    "    fc.ExtractSchemaField(\n",
    "        name=\"component\",\n",
    "        data_type=fc.StringType,\n",
    "        description=\"Affected component or system\"\n",
    "    )\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error Log Classification and Extraction\n",
    "\n",
    "This cell performs semantic analysis on the error log dataset. It classifies the severity of each error log (e.g., low, medium, high, critical) and extracts key debugging information, such as the root cause and recommended fix, using a language model. \n",
    "\n",
    "The results are displayed in a readable table, showing the timestamp, service, severity, root cause, and fix recommendation for each error entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Analyze errors using semantic operations\n",
    "df_analyzed = df.select(\n",
    "        \"timestamp\",\n",
    "        \"service\",\n",
    "        # Classify error severity\n",
    "        fc.semantic.classify(\"error_log\", [\"low\", \"medium\", \"high\", \"critical\"]).alias(\"severity\"),\n",
    "        # Extract key debugging information\n",
    "        fc.semantic.extract(\n",
    "            \"error_log\",\n",
    "            error_analysis_schema\n",
    "        ).alias(\"analysis\")\n",
    "    )\n",
    "\n",
    "# Show analysis with extracted fields\n",
    "df_analysis_readable = df_analyzed.select(\n",
    "    \"timestamp\",\n",
    "    \"service\",\n",
    "    \"severity\",\n",
    "    df_analyzed.analysis.root_cause.alias(\"root_cause\"),\n",
    "    df_analyzed.analysis.fix_recommendation.alias(\"fix_recommendation\")\n",
    ")\n",
    "\n",
    "print(\"Error Analysis Results:\")\n",
    "print(\"-\" * 70)\n",
    "df_analysis_readable.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Displaying Critical Errors\n",
    "\n",
    "This cell filters the analyzed error logs to focus on entries classified as \"critical\" or \"high\" severity. \n",
    "\n",
    "It then displays a table with the timestamp, service, root cause, and recommended fix for each of these urgent errors, helping to quickly identify and prioritize issues that require immediate attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Focus on critical errors\n",
    "print(\"\\nCritical Errors Requiring Immediate Attention:\")\n",
    "print(\"-\" * 70)\n",
    "critical_errors = df_analyzed.filter(\n",
    "    (df_analyzed[\"severity\"] == \"critical\") | (df_analyzed[\"severity\"] == \"high\")\n",
    ").select(\n",
    "    \"timestamp\",\n",
    "    \"service\",\n",
    "    df_analyzed.analysis.root_cause.alias(\"root_cause\"),\n",
    "    df_analyzed.analysis.fix_recommendation.alias(\"fix_recommendation\")\n",
    ")\n",
    "\n",
    "critical_errors.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting and Displaying Error Patterns\n",
    "\n",
    "This cell extracts structured error patterns from each log entry, identifying the type of error (such as connection issues or exceptions) and the affected component or system. \n",
    "\n",
    "It then displays a table summarizing these patterns for each service, providing a clear overview of recurring error types and their sources across the application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract specific error patterns\n",
    "\n",
    "df_patterns = df.select(\n",
    "        \"service\",\n",
    "        fc.semantic.extract(\n",
    "            \"error_log\",\n",
    "            error_pattern_schema\n",
    "        ).alias(\"patterns\")\n",
    "    )\n",
    "\n",
    "print(\"\\nError Patterns Detected:\")\n",
    "print(\"-\" * 70)\n",
    "\n",
    "# Show pattern details\n",
    "df_pattern_details = df_patterns.select(\n",
    "        \"service\",\n",
    "        df_patterns.patterns.error_type.alias(\"error_type\"),\n",
    "        df_patterns.patterns.component.alias(\"component\")\n",
    "    )\n",
    "df_pattern_details.show()\n",
    "\n",
    "# Clean up\n",
    "session.stop()\n",
    "\n",
    "print(\"\\nAnalysis complete!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "- Try adding your own error logs\n",
    "- Extract specific fields like error codes or user IDs\n",
    "- Build alerts for critical errors\n",
    "- Create auto-generated runbooks"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
